Skip to content

Conversation

@yhakbar
Copy link
Collaborator

@yhakbar yhakbar commented Oct 28, 2025

Description

Starts integrating filters into the logic for discovery so that we can prevent discovery of components when certain filters are set.

TODOs

Read the Gruntwork contribution guidelines.

  • I authored this code entirely myself
  • I am submitting code based on open source software (e.g. MIT, MPL-2.0, Apache)]
  • I am adding or upgrading a dependency or adapted code and confirm it has a compatible open source license
  • Update the docs.
  • Run the relevant tests successfully, including pre-commit checks.
  • Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added / Removed / Updated [X].

Migration Guide


Note

Integrates filter-aware discovery (with experiment flag), refactors constructor, and renames filter parsing requirements to "RequiresDiscovery" with updated errors and tests.

  • Discovery:
    • Enable filter-aware traversal via WithFilterFlagEnabled and WithFilters; positive filters set excludeByDefault.
    • Skip exclude globs during walk when filter flag is enabled; early file-level filtering and refined hidden-dir handling.
    • Extract NewDiscovery to constructor.go; wire filters into NewForDiscoveryCommand and NewForHCLCommand.
    • Add createComponentFromPath helper; minor concurrency/walk refactors.
  • Filter Engine:
    • Rename RequiresHCLParsingRequiresDiscovery across AST and Filters.
    • Replace FilterQueryRequiresHCLParsingError with FilterQueryRequiresDiscoveryError and update messages.
  • Tests:
    • Add/extend discovery and integration tests for filter behavior (include/exclude defaults, reading file filters incl. absolute paths) and updated error types.

Written by Cursor Bugbot for commit cad37ef. This will update automatically on new commits. Configure here.

Summary by CodeRabbit

  • New Features

    • Filters are now evaluated during the discovery phase, enabling more efficient filtering of Terragrunt configurations.
    • Added improved handling for symlinked paths in filter-based discovery.
  • Bug Fixes

    • Enhanced directory exclusion logic to respect filter configurations more accurately.
  • Tests

    • Added test coverage for filter-driven discovery behavior.

@vercel
Copy link

vercel bot commented Oct 28, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
terragrunt-docs Ready Ready Preview Comment Nov 6, 2025 1:57am

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 28, 2025

📝 Walkthrough

Walkthrough

The PR refactors discovery constructor logic to conditionally enable and parse filters when the FilterFlag experiment is active, introduces a new public NewDiscovery constructor with default worker/directory initialization, and renames filter interface methods from RequiresHCLParsing to RequiresDiscovery along with associated error types throughout the filter package.

Changes

Cohort / File(s) Summary
Discovery Constructor Restructuring
internal/discovery/constructor.go
Reworked filter handling logic when FilterFlag experiment is enabled to parse and apply filters only if present; moved filter parsing behind explicit length checks; introduced new public NewDiscovery(dir string, opts ...DiscoveryOption) constructor that computes default worker count, initializes Discovery with include/exclude directories, and applies optional functional options.
Discovery Core Logic
internal/discovery/discovery.go
Removed old NewDiscovery constructor and runtime import; added filterFlagEnabled field with WithFilterFlagEnabled() method; updated WithFilters to enable filter flag and conditionally set excludeByDefault; introduced createComponentFromPath helper; enhanced processing flow to bypass exclude patterns when filter flag is enabled; updated shouldSkipDirectory and path handling to respect filter flag; added isInHiddenDirectory helper for filter-driven discovery.
Discovery Tests
internal/discovery/discovery_test.go, internal/discovery/filter_integration_test.go
Added new test TestDiscoveryExcludesByDefaultWhenFilterFlagIsEnabled verifying filter flag behavior with various filter combinations; added internal/filter import; added symlink evaluation (filepath.EvalSymlinks) for tmpDir in filter integration test.
Filter Package Interface Refactoring
internal/filter/ast.go, internal/filter/filters.go
Renamed RequiresHCLParsing() method to RequiresDiscovery() across Expression interface and all implementations (PathFilter, AttributeFilter, PrefixExpression, InfixExpression); updated method call sites in Filters type.
Filter Error Type Refactoring
internal/filter/errors.go
Renamed public error type from FilterQueryRequiresHCLParsingError to FilterQueryRequiresDiscoveryError with updated error message and documentation to reference discovery terminology.
Integration Tests
test/integration_hcl_filter_test.go
Updated test expectations to use FilterQueryRequiresDiscoveryError instead of FilterQueryRequiresHCLParsingError in filter-related test cases.

Sequence Diagram

sequenceDiagram
    participant Discovery
    participant Filter
    participant Component Creation
    
    Note over Discovery: filterFlagEnabled = true
    Discovery->>Discovery: processFile(path)
    alt filterFlagEnabled is true
        Discovery->>Discovery: Skip exclude pattern checks
        Discovery->>Component Creation: createComponentFromPath()
        alt Hidden directory allowed
            Component Creation->>Filter: Evaluate filters
            alt Filters match
                Filter-->>Component Creation: ✓ Include
                Component Creation-->>Discovery: Return component
            else Filters don't match
                Filter-->>Component Creation: ✗ Skip
                Component Creation-->>Discovery: Return nil
            end
        else Hidden directory not allowed
            Component Creation-->>Discovery: Skip component
        end
    else filterFlagEnabled is false
        Discovery->>Discovery: Apply standard exclude patterns
        Discovery->>Component Creation: Regular component creation
    end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Discovery constructor restructuring: Verify filter conditional logic is correctly restructured and default initialization (worker count, directory paths) is sound in new NewDiscovery constructor
  • Filter-flag control flow: Carefully review the logic in discovery.go that bypasses exclude patterns when filterFlagEnabled is true, including the createComponentFromPath path and shouldSkipDirectory changes, to ensure hidden directory handling and filter evaluation order is correct
  • Interface rename consistency: Verify that all call sites across discovery and filter packages have been updated to use RequiresDiscovery() instead of RequiresHCLParsing() and that error type references are complete
  • Test coverage: Confirm new test TestDiscoveryExcludesByDefaultWhenFilterFlagIsEnabled properly exercises the filter-flag behavior with various filter combinations

Possibly related PRs

Suggested reviewers

  • denis256

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat: Integrating filters into discovery' accurately reflects the main objective—integrating filter logic into the discovery system to prevent component discovery based on filter settings.
Description check ✅ Passed The description is complete with clear explanation of changes, confirmed TODOs (submitting based on open source, docs updated, tests run, release notes included), but lacks a substantive release notes entry and migration guide details.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/integrating-filters-into-discovery

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2b95c4e and cad37ef.

📒 Files selected for processing (8)
  • internal/discovery/constructor.go (3 hunks)
  • internal/discovery/discovery.go (6 hunks)
  • internal/discovery/discovery_test.go (2 hunks)
  • internal/discovery/filter_integration_test.go (1 hunks)
  • internal/filter/ast.go (5 hunks)
  • internal/filter/errors.go (1 hunks)
  • internal/filter/filters.go (2 hunks)
  • test/integration_hcl_filter_test.go (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.go

⚙️ CodeRabbit configuration file

Review the Go code for quality and correctness. Make sure that the Go code follows best practices, is performant, and is easy to understand and maintain.

Files:

  • test/integration_hcl_filter_test.go
  • internal/filter/filters.go
  • internal/discovery/filter_integration_test.go
  • internal/discovery/constructor.go
  • internal/filter/ast.go
  • internal/discovery/discovery.go
  • internal/filter/errors.go
  • internal/discovery/discovery_test.go
🧠 Learnings (3)
📚 Learning: 2025-08-19T16:05:54.723Z
Learnt from: Resonance1584
Repo: gruntwork-io/terragrunt PR: 4683
File: go.mod:86-90
Timestamp: 2025-08-19T16:05:54.723Z
Learning: When analyzing Go module dependencies for removal, always check for both direct imports and API usage across all Go files in the repository, not just a quick search. The github.com/mattn/go-zglob library is used for filesystem walking and glob expansion in multiple Terragrunt files including util/file.go, format commands, and AWS provider patch functionality.

Applied to files:

  • internal/discovery/constructor.go
  • internal/discovery/discovery_test.go
📚 Learning: 2025-02-10T13:36:19.542Z
Learnt from: levkohimins
Repo: gruntwork-io/terragrunt PR: 3723
File: cli/commands/stack/action.go:160-160
Timestamp: 2025-02-10T13:36:19.542Z
Learning: The project uses a custom error package `github.com/gruntwork-io/terragrunt/internal/errors` which provides similar functionality to `fmt.Errorf` but includes stack traces. Prefer using this package's error functions (e.g., `errors.Errorf`, `errors.New`) over the standard library's error handling.

Applied to files:

  • internal/discovery/constructor.go
  • internal/discovery/discovery_test.go
📚 Learning: 2025-07-03T22:05:07.356Z
Learnt from: wakeful
Repo: gruntwork-io/terragrunt PR: 4491
File: test/integration_test.go:4265-4265
Timestamp: 2025-07-03T22:05:07.356Z
Learning: Constants defined in one Go file are accessible from other files in the same package without explicit imports. When reviewing Go code, consider the package-level scope rather than just the individual file scope.

Applied to files:

  • internal/discovery/discovery.go
🧬 Code graph analysis (5)
test/integration_hcl_filter_test.go (1)
internal/filter/errors.go (1)
  • FilterQueryRequiresDiscoveryError (49-51)
internal/filter/filters.go (2)
internal/filter/ast.go (1)
  • Expression (11-18)
internal/filter/errors.go (1)
  • FilterQueryRequiresDiscoveryError (49-51)
internal/discovery/constructor.go (4)
internal/experiment/experiment.go (2)
  • Experiments (46-46)
  • FilterFlag (36-36)
internal/filter/filters.go (1)
  • ParseFilterQueries (19-46)
internal/discovery/discovery.go (2)
  • DiscoveryOption (162-162)
  • Discovery (81-159)
config/stack.go (1)
  • StackDir (35-35)
internal/discovery/discovery.go (5)
pkg/log/log.go (1)
  • Debugf (72-74)
util/file.go (1)
  • CleanPath (642-644)
config/stack.go (1)
  • StackDir (35-35)
internal/filter/evaluator.go (1)
  • Evaluate (23-29)
internal/component/component.go (2)
  • Components (302-302)
  • Component (27-41)
internal/discovery/discovery_test.go (4)
options/options.go (1)
  • NewTerragruntOptions (380-382)
test/helpers/logger/logger.go (1)
  • CreateLogger (9-14)
internal/filter/filters.go (1)
  • ParseFilterQueries (19-46)
internal/component/component.go (1)
  • UnitKind (21-21)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
  • GitHub Check: Pull Request has non-contributor approval
🔇 Additional comments (20)
test/integration_hcl_filter_test.go (1)

123-147: LGTM! Error type updates are consistent.

The test expectations have been correctly updated to use FilterQueryRequiresDiscoveryError instead of FilterQueryRequiresHCLParsingError, aligning with the broader refactoring toward discovery-based terminology.

internal/discovery/filter_integration_test.go (1)

811-812: LGTM! Symlink resolution improves path consistency.

Resolving symlinks in the temporary directory ensures that absolute path comparisons work correctly during filter evaluation, which is important for accurate test results.

internal/discovery/discovery_test.go (1)

659-723: LGTM! Well-structured test for filter-flag behavior.

The new test comprehensively validates the filter-flag-enabled discovery behavior, including default inclusion/exclusion logic with positive and negative filters. The symlink resolution ensures path consistency, and the test follows established patterns in the codebase.

internal/discovery/constructor.go (3)

124-144: LGTM! Well-designed public constructor with sensible defaults.

The NewDiscovery constructor properly initializes discovery with:

  • Dynamic worker count bounded between 4 and 8 based on CPU cores
  • Default inclusion of stack directory components
  • Default exclusion patterns for .git, .terraform, .terragrunt-cache
  • Support for functional options pattern via DiscoveryOption

88-99: LGTM! Proper filter integration with experiment flag.

The filter handling correctly:

  • Guards filter functionality behind the FilterFlag experiment
  • Only parses and applies filters when queries are provided
  • Enables the filter flag on discovery before setting filters
  • Returns errors appropriately if parsing fails

108-119: LGTM! Consistent filter handling pattern.

The HCL command constructor follows the same pattern as the discovery command constructor for filter handling, ensuring consistency across the codebase.

internal/filter/errors.go (1)

48-58: LGTM! Clear terminology update.

The rename from FilterQueryRequiresHCLParsingError to FilterQueryRequiresDiscoveryError better reflects the actual requirement and aligns with the broader refactoring toward discovery-based terminology. The error message is also clearer.

internal/filter/filters.go (2)

59-68: LGTM! Clear method rename with improved return statement.

The rename from RequiresHCLParsing to RequiresDiscovery aligns with the discovery-based terminology. The explicit return nil, false at line 67 improves code clarity.


117-119: LGTM! Consistent error type usage.

Correctly uses RequiresDiscovery() and returns the updated FilterQueryRequiresDiscoveryError type.

internal/filter/ast.go (5)

16-17: LGTM! Interface method rename aligns with discovery terminology.

The rename from RequiresHCLParsing to RequiresDiscovery is consistent with the broader refactoring.


51-53: LGTM! Correct implementation.

PathFilter correctly returns false for RequiresDiscovery() since path-based filtering doesn't require component discovery.


96-98: LGTM! Correct implementation.

AttributeFilter correctly returns true for RequiresDiscovery() since attribute-based filtering requires component discovery and parsing.


108-110: LGTM! Proper delegation.

PrefixExpression correctly delegates to its Right expression to determine if discovery is required.


123-132: LGTM! Correct composite logic.

InfixExpression properly checks both Left and Right expressions, returning true if either requires discovery, and (nil, false) only when neither does.

internal/discovery/discovery.go (6)

343-360: LGTM! Correct filter integration logic.

The WithFilters method properly:

  • Enables the filter flag when filters are set
  • Sets excludeByDefault = true when positive filters exist (correct semantics: "only include what matches")
  • The comment clearly explains the optimization rationale

362-367: LGTM! New public method for filter flag control.

The WithFilterFlagEnabled method provides explicit control over the filter flag experiment state, which is useful for testing and conditional feature enablement.


681-684: LGTM! Correct behavior for filter-driven discovery.

When the filter flag is enabled, returning nil (don't skip) allows the discovery to walk the entire directory tree and defer component selection to filter evaluation, which is the correct approach.


734-785: LGTM! Well-structured filter-driven discovery path.

The new filter-flag-enabled path (lines 745-785) is well-designed:

  • Creates components early via createComponentFromPath
  • Properly handles hidden directories with .terragrunt-stack exception
  • Implements early filter evaluation when dependencies aren't being discovered
  • Clean separation from legacy code path

The logic flow is sound and the implementation is clear.


841-863: LGTM! Clean helper function for component creation.

The createComponentFromPath function properly:

  • Iterates through configured filenames to find matches
  • Creates Stack vs Unit based on the matched filename
  • Sets the discovery context when available
  • Returns nil when no filename matches (correct fallback)

787-789: Good practice documenting temporary code.

The comment clearly marks the legacy code path and indicates it should be removed once the filter flag is generally available, which helps future maintainers understand the transition period.


Comment @coderabbitai help to get the list of available commands and usage tips.

@yhakbar yhakbar force-pushed the chore/disable-experimental-ci-ignore branch 3 times, most recently from c574388 to 4de37b3 Compare October 30, 2025 16:01
@yhakbar yhakbar force-pushed the feat/integrating-filters-into-discovery branch from d4e5a07 to 474e9ed Compare October 30, 2025 16:58
@yhakbar yhakbar force-pushed the chore/disable-experimental-ci-ignore branch 3 times, most recently from b684454 to 7c30c1b Compare November 4, 2025 23:34
@yhakbar yhakbar force-pushed the feat/integrating-filters-into-discovery branch from 845b4f8 to 652c6dc Compare November 4, 2025 23:40
Base automatically changed from chore/disable-experimental-ci-ignore to main November 5, 2025 13:34
chore: Renaming `RequiresHCLParsing` to `RequiresDiscovery` as that captures the essence of the check better

chore: Testing exclude by default with filters
@yhakbar yhakbar force-pushed the feat/integrating-filters-into-discovery branch from 652c6dc to cad37ef Compare November 6, 2025 01:56
@yhakbar yhakbar marked this pull request as ready for review November 6, 2025 11:10
@yhakbar yhakbar requested a review from denis256 as a code owner November 6, 2025 11:10
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

return nil
}
}
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Unnecessary Filter Evaluation on Empty Filters Flag

Missing check for empty filters before evaluation. When d.filterFlagEnabled is true but d.filters is empty (e.g., when filter flag experiment is enabled but no filters are provided), the code at line 771 will still attempt to evaluate filters unnecessarily. The condition if _, requiresParsing := d.filters.RequiresDiscovery(); !requiresParsing will be true when filters is empty (returns false), causing d.filters.Evaluate() to be called on every component even when there are no filters to apply. This is inconsistent with the pattern used elsewhere in the code (lines 1041 and 1120) where len(d.filters) > 0 is checked before evaluation. While not causing incorrect behavior (empty filters return input unchanged), this creates unnecessary overhead during file processing.

Fix in Cursor Fix in Web

cleanDir := util.CleanPath(canonicalDir)
for part := range strings.SplitSeq(cleanDir, "/") {
if part == config.StackDir {
allowHidden = true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code can be simplified into separate function called isInStackDirectory(path)

@yhakbar yhakbar merged commit b9cbc8c into main Nov 6, 2025
141 of 145 checks passed
@yhakbar yhakbar deleted the feat/integrating-filters-into-discovery branch November 6, 2025 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants